Internet search engines allow computer users to use their Internet browsers (e.g., Mozilla Firefox) to submit search query terms to those search engines by entering those query terms into a search field (also called a “search box”). After receiving query terms from a user, an Internet search engine determines a set of Internet-accessible resources that are pertinent to the query terms, and returns, to the user's browser, as a set of search results, a list of the resources most pertinent to the query terms, usually ranked by query term relevance.
These resources are often individual web pages or web sites. Each search result item in a list of search result items may specify a title of a web page or web site, an abstract for that web page or web site, and a hyperlink which, when selected or clicked by the user, causes the user's browser to request the web page (or a web page from the web site) over the Internet.
Some Internet search engines also provide mechanisms through which a user can specify that he is interested only in search results that are images. Under such circumstances, in response to the user's submission of the query terms, the search engine may search a corpus of word-tagged images for images that have been tagged with words that match one or more of the query terms. The search engine may then provide, to the user, a search results web page that contains thumbnail versions of the matching images that the search engine found. Unfortunately, these matching images might in some cases actually have very little to do with the query terms that the user entered. The manner in which the words associated with the images become associated with those images is often highly subjective in nature.
For example, an image might have been tagged with several different words by many different users. To each of these different users, the image might seem to be of a different thing. One user might see an image of a mountain in a particular image, and might tag that particular image generically with the word “mountain.” Another user might recognize (or mistakenly think) that the mountain in the image is actually in a specific national park, and so that user might tag the same particular with the actually name of the national park (which might or might not be correct). Yet another user, who took the photograph that resulted in the particular image, might recall that the mountain was seen while he was hiking with a certain friend of his on a certain date. That user might then tag the same particular image with the name of his friend, the date on which the photo was taken, and some topical indication like “hiking trip.” Most search engines are only able to search images based on textual metadata that largely consists of tags, such as the ones discussed above, attached to those images by various viewers of those images. In some cases, the quantity of metadata attached to an image might be small; such an image might not be produced as a result of many searches.
As a result of the usually highly subjective nature of the searchable metadata that is associated with each image in a searchable corpus of images, the results for a search for an image that matches user-specified query terms can produce a wide variety of images, many of which might not relate in the slightest bit to the topic imagined by the user who specified the query terms. If the user is lucky enough to find, in the search results page, an image in which he is interested, then the user might want to view other images in the search corpus which are deemed to be similar, in some way, to the image in which the user actually was interested. Some Internet search engines display, on the search results web page, a control through which a user can request, from the search engine, additional images that are similar to an image that the user selects from the initial set of image search results. However, automatically determining whether one image actually is “similar” to another image, according to some aspect that the user imagines but does not necessarily express to the search engine, can be a challenging task.
Because existing search engines do not have any sophisticated mechanism for determining what a user really means when the user indicates that he wants to see other image search results that are similar to a selected image, existing search engines often will return, to a user has expressed such a desire, images that are not really similar to the selected image, or images that could be similar to the selected image only in ways about which the user does not really care. Existing search engines' approaches for determining which images are similar to other images, in a way that is likely to satisfy a searching user's desires, are often woefully inadequate. One approach for determining whether two images are similar involves creating, for each of the images, a numerical vector that reflects visual features of that image (such as the number of colors in the image), and then comparing the vectors to each other. If the vectors are similar enough to each other, then the images to which those vectors correspond are also deemed to be similar to each other. Images that have vectors that are similar to each other are deemed to be more similar to each other than images that have vectors that are not similar to each other. In reality, though, even images that have vectors that are similar to each other often will not really pertain to similar content. For example, images that have a similar color palette might pertain to subjects that are not related to each other in any imaginable way (other than being similar in color).
One weakness that plagues existing approaches to determining image similarity is that those approaches are not designed to account for what the user might or might not know about the subject of the image. Often, what the user knows about the subject of the image will be very relevant to what the user wants to see when the user asks to see images that are similar to some selected image. For example, a user might ask to see images that are similar to an image of U.S. President Barack Obama playing in the surf. If the user knows that the person playing in the surf in the image is Barack Obama, then the user probably is interested in seeing more images that pertain to Barack Obama, regardless of whether those images show Barack Obama playing in the surf. However, if the user does not know that the person playing in the surf in the image is Barack Obama, then the user probably is interested in seeing more images of people playing in the surf, regardless of whether those images show Barack Obama at all. Existing approaches for determining whether images are similar do not take into account, at all, what the user might or might not know about the subject of the image for which the user is seeking to find similar images.
The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.